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ratio  scaling  [the  analytic  hierarchy  process  proposed  by  Saaty  (1977,  1980), 
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The  development  of  analytical  procedures  and  experimental  techniques  for 
constructing  ratio  measurement  scales  has  long  been  a  major  topic  and  important 
challenge  in  psychophysics  and  other  areas  of  psychology.  The  interest  in  ratio 
scales  is  obviously  related  to  their  high  level  of  invariance  (unique  up  to  a 
similarity  transformation)  and  the  associated  statistical  operations  they  allow 
one  to  perform  (Stevens,  1946).  To  obtain  ratio  scales,  Stevens  and  his  colleagues 
(Stevens,  Mack,  &  Stevens,  1960)  developed  the  "cross  modality  matching"  paradigm, 
which  was  originally  applied  to  variables  with  a  corresponding  physical  continuum 
and  later  generalized  to  social  and  other  psychological  stimuli  (e.g..  Lodge, 

1981) .  Although  Stevens '  techniques  have  been  widely  used  in  numerous  areas 
(e.g.,  Stevens,  1972,  1975;  Lodge,  1981),  their  precise  characterization  is 
still  a  topic  of  debate  among  measurement  specialists  (e.g.,  Shepard,  1981). 

Following  two  decades  of  development  of  ordinal  level  scales,  which  culmi¬ 
nated  in  the  development  of  nonmetric  multidimensional  scaling,  there  has  recently 
been  a  renewed  interest  in  the  problem  of  ratio  scale  measurement.  To  a  large 
extent,  the  interest  in  this  problem  is  due  to  the  development  of  a  new  ratio 
scaling  procedure  and  its  successful  use  in  a  variety  of  experimental  and 
practical  situations.  The  most  salient  characteristic  of  this  method  was 
apparently  first  discovered  by  Gulliksen  (1959),  but  the  procedure  was  fully 
developed,  investigated,  described,  and  applied  by  Saaty  (1977,  1980). 

Saaty's  Analytic  Hierarchy  Process 

Consider  a  set  of  n  stimuli  with  unknown  scale  values,  s1  =  (s,,...,s  ). 

—  1  n 

Following  a  process  of  pairwise  comparisons,  an  n  x  n  matrix  of  ratios,  R,  is 
constructed  such  that 

(1)  r_^  j  *  si/Sj  ,  i ,  j s  1 , « •  •  n , 

All  the  entries  in  R  are  positive,  satisfying  the  reciprocity  condition: 

r  .  =  1/r  .  In  reality,  this  constraint  is  artificially  enforced  because 

J  » i  J  *  i 

typically  only  n(n-l)/2  judgments  are  obtained  and  the  remaining  ratios  are 

calculated  by  the  reciprocal  transformation  and  by  assuming  that  r  =  1 

*  * 

(i*l, . . . ,n) . 

Saaty's  procedure  is  based  on  a  relatively  simple  and  elementary  property 
of  R:  when  postmult iplied  by  the  vector  of  scale  values,  the  result  is  a  vector 


related  to  the  scale  values  by  a  constant.  This  constant  turns  out  to  be  the 
size  of  the  matrix  R.  Thus, 


(2) 


Rs  *  ns . 


This  formulation  suggests  that  the  unknown  scale  values  can  be  obtained  by  an 
eigenvalue-eigenvector  decomposition  of  R.  Saaty's  solution  to  this  problem 
when  the  data  are  perturbed  is  to  use  the  normalized  right  eigenvector  associated 
with  the  largest  eigenvalue  of  R,  denoted  by  X^^,  as  an  estimate  of  the  scale 
values . 


The  matrix  R  is  said  to  be  cons istent  if  the  ratios  r  ,  satisfy 

1  *  J 


(3) 


r ,  ,  r, 
i,k  k,j 


i, j ,k=l, . . .n 


for  all  i,  j,  and  k.  A  consistent  matrix  is  of  unit  rank.  Moreover,  its  only 

nonzero  eigenvalue  must  be  n  (see  Eq .  2).  It  can  be  shown  that  X  >  n  when 

max 

consistency  is  violated.  Therefore,  Saaty  recommends  using  the  normalized 
difference 


(4)  y  =  (X  -n)/(n-l) 

max 

as  a  measure  of  inconsistency.  If  R  is  consistent,  y  =  0;  if  not,  y  is  raonotoni- 
cally  increasing  in  the  magnitude  of  departure  from  consistency  (for  more  details 
of  the  properties  of  this  method  see  Saaty,  1977,  1980). 

Saaty's  analytic  hierarchy  process  (AHP)  has  attracted  much  attention,  having 
been  successfully  applied  in  such  diverse  areas  as  marketing  (Wind  &  Saaty,  1980), 
political  science  (Saaty  &  Bennett,  1977),  and  the  measurement  of  subjective 
probabilities  (Yager,  1979).  However,  the  AHP  has  had  its  share  of  criticism. 

In  particular,  Johnson,  Beine,  and  Wang  (1979)  have  pointed  out  that  for  inconsis¬ 
tent  matrices  of  order  n  _>  4  the  solution  is  not  invariant  under  transposition. 

In  other  words,  the  right  eigenvector  of  R'  (i.e.,  the  left  eigenvector  of  R)  is 
not  necessarily  the  reciprocal  of  the  right  eigenvector  of  R.  This  may  cause 
difficulties  in  the  interpretation  of  the  scale  values  (see  also  Budescu,  1984). 

The  Geometric  Means  Procedure 

Another  criticism  was  leveled  by  Williams  and  Crawford  (1980),  who  argued 
that,  unlike  most  estimation  procedures,  Saaty's  procedure  does  not  optimize  a 
well  defined  loss  function.  Moreover,  it  requires  complex  calculations  as  the 
maximal  eigenvalue  and  the  associated  eigenvector  are  calculated  by  an  iterative 
technique.  Therefore,  Williams  and  Crawford  proposed  an  alternative  procedure. 


called  the  geometric  means  procedure,  which  is  invariant  under  transposition 
and  can  easily  be  calculated  by  hand. 

It  has  long  been  recognized  that  the  geometric  means  (GMs)  are  least 
squares  estimates  of  the  logarithms  of  the  scale  values  (e.g.,  Torgerson,  1958). 
Williams  and  Crawford  (1980)  have  shown  that  if  the  true  scores  are  perturbed 
by  independent  lognormally  distributed  errors  with  zero  mean  and  variance  a2, 
the  GMs  are  also  maximum  likelihood  estimates  of  the  scale  values  (for  a  theore¬ 
tical  justification  of  the  lognormal  distribution  of  errors  in  the  judgment  of 
dissimilarities  see  Ramsay,  1977).  To  measure  the  inconsistency  of  R,  Williams 
and  Crawford  proposed  using  the  residual  mean  square 
n  n 

(5)  S2  =  I  £  [£n(r  . ) -£n(GM  /GM. ) ] 2/ (n-1) (n-2) , 

i=l  j  =  l  2 

where  S2  is  an  estimate  of  a2.  As  in  Eq.  4,  S2  =  0  if  R  is  consistent.  As  the 
departure  from  consistency  increases,  S2  increases  monotonically . 

A  Comparison  Based  on  Perturbed  Scale  Values 

How  do  the  two  scaling  procedures  compare?  For  a  consistent  judgment 
matrix  of  any  order  or  for  any  reciprocal  matrix  of  order  n  £  3,  the  eigenvector 
of  R  (associated  with  ^max)  and  the  vector  of  row  GMs  are  equal  except  for  a 
similarity  transformation.  The  procedures  differ,  however,  when  consistency 
is  violated.  To  compare  the  two  procedures,  Williams  and  Crawford  conducted 
a  Monte  Carlo  study  in  which  reciprocal  matrices  of  order  5,  7,  and  10  were 
perturbed  by  multiplying  each  ratio  S^/S^  by  an  error  eij-  Two  types  of  errors 
were  investigated.  In  the  first,  the  errors  e^  were  drawn  from  a  lognormal 
distribution  with  zero  mean  and  variance  a2.  In  the  second,  the  errors  were 
drawn  from  a  population  of  ratios  of  uniform  random  variables  with  the  same 
mean  and  variances  as  in  the  first  case.  Five  values  of  o2  were  examined,  and 
each  condition  was  repeated  1000  times.  In  all  cases  the  GM  procedure  outper¬ 
formed  Saaty's  procedure  according  to  both  the  least  squares  and  the  log  least 
squares  criteria.  Moreover,  the  relative  advantage  of  the  GM  procedure  increased 
with  both  the  size  of  the  judgment  matrix  R  and  the  variability  of  error. 

Table  1,  which  is  taken  from  Tables  2  and  3  of  Williams  and  Crawford,  shows  the 
percentage  of  cases  (matrices)  in  which  the  GM  procedure  outperformed  the  AHP. 

In  the  "almost  consistent  case"  (top  row  of  Table  1),  when  the  variability 
of  error  is  very  small,  the  two  solutions  are  equally  good.  But  as  the  varia¬ 
bility  of  error  and  the  size  of  the  response  matrix  increase,  the  GM's  advantage 
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Table  1 


Percentage  of  Cases  in  which  the  GM  Procedure  Provides 
Better  Least  Squares  Estimates  of  Scale  Values  than  the  AHP 


Lognormal  Errors  Ratio  of  Uniform  Errors 


02 

5 

n 

7 

10 

Total 

5 

n 

7 

10 

Total 

.01 

52 

51 

50 

51.00 

51 

49 

50 

50.00 

.16 

54 

56 

57 

55.67 

51 

52 

53 

52.00 

.64 

50 

54 

57 

53.67 

57 

62 

66 

61.67 

1.00 

56 

62 

70 

62.67 

59 

68 

73 

66.67 

1.69 

58 

66 

71 

65.00 

71 

81 

92 

81.33 

Total 

54.00 

57.80 

61.00 

57.60 

57.80 

62.40 

66.80 

62.33 

Note : 

Each  cell 

is  based 

on  1000 

cases . 

over  Saaty's  procedure  is 

enhanced 

.  Table  1 

suggests 

that  nothing  is  lost  by 

using  the  GM  procedure  in 

cases  where  Saaty' 

s  AHP  works  well. 

There  seems  to  be 

a  real 

potential 

gain  in  employing 

the  GM  procedure  in 

other  more  erratic  cases. 

A  particularly  attractive  feature  of  Saaty's  AHP  is  the  availability  of  a 
consistency  index  y  (Eq.  4)  which  differentiates  between  judgment  matrices  that 
can  be  maintained  and  interpreted,  and  judgment  matrices  that  must  be  rejected 
as  "randomly  generated  data."  As  a  rule  of  thumb,  Saaty  (1977)  proposed  to  fix 
y  at  0.1.  Later  (1980,  1983),  he  recommended  using  a  consistency  ratio  denoted 
by  C.R.  and  defined  by 

(6)  C.R.  =  y/y  , 

n 

where  y^  is  an  empirical  measure  computed  from  Eq.  (4)  for  500  randomly  generated 
reciprocal  matrices  of  order  n.  The  new  rule  of  thumb  (presumably  for  1-9  re¬ 
sponse  scales)  is  to  accept  only  judgment  matrices  for  which  C.R.  £  0.2. 

Although  Williams  and  Crawford  have  suggested  the  statistic  S2  as  a  measure 
of  inconsistency  for  the  GM  procedure,  no  corresponding  rules  are  available  for 
its  use.  This,  perhaps,  reflects  the  shortage  of  applications  of  the  GM  proce¬ 
dure.  More  importantly,  because  y  and  S2  are  based  on  two  different  models  of 
scaling  matrices  of  ratio  judgments,  and  do  not  necessarily  reflect  the  same 
properties  of  the  data,  it  is  still  an  open  question  how  the  two  procedures  compare 
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in  the  null  case — when  the  data  are  known  to  be  random  (rather  than  consistent 
judgments  perturbed  by  error). 

Objectives 

A  major  purpose  of  the  present  note  is  to  contrast  and  evaluate  the  two 
scaling  procedures  in  the  null  case  and  provide  guidelines  for  their  use  with 
real  data.  In  this  case  both  scaling  procedures  are  expected  to  reject  by 
means  of  M  and  S  the  null  hypothesis  of  consistency.  The  methodology  employed 
in  the  present  paper  is  very  similar  to  that  used  by  Saaty  (1977,  1980):  a 
large  number  of  reciprocal  matrices  consisting  of  random  entries  are  generated, 
solutions  and  indices  of  inconsistency  are  calculated,  and  the  rejection  rules 
are  compared  to  each  other. 

A  second  purpose  of  this  note  is  to  generalize  the  findings  from  the 
comparison  of  the  two  scaling  procedures  to  a  nonnumerical  method  of  obtaining 
ratio  judgments.  Saaty's  procedure  obtains  directly  numerical  estimates  of 
ratios  in  the  tradition  of  Stevens'  magnitude  estimation.  Furthermore,  Saaty 
strongly  recommends  restricting  the  response  scale  to  the  positive  integers 
1  through  9  and  their  reciprocals  (a  total  of  17  possible  different  values). 
Another  equally  popular  experimental  procedure  for  eliciting  ratio  judgments 
is  the  "constant  sum"  method  (e.g.,  Torgerson,  1958)  in  which  a  judge  is 
required  to  divide  a  constant  number  of  units  between  two  stimuli  in  accordance 
with  their  ratio.  For  example,  a  90:10  allocation  of  100  units  reflects  a 
ratio  of  90/10  =  9,  and  a  50:50  division  gives  rise  to  a  ratio  of  1.  In  cer¬ 
tain  applications,  such  as  encoding  subjective  probabilities,  it  is  preferable 
to  avoid  using  numerical  responses  because  of  their  inherent  bias.  Instead, 
graphical  methods  such  as  placing  a  sliding  marker  along  a  bounded  straight 
line  or  adjusting  a  two-color  probability  wheel  are  recommended  (e.g.,  Spetzler 


&  Stael  von  Holstein,  1975;  Wallsten  &  Budescu,  1983).  Then  the  ratio  judg¬ 
ments  can  be  derived  directly  from  the  relative  lengths  (areas)  of  the  two 
segments  of  the  line  (wheel)  as  in  the  direct  estimation  method. 

When  graphical  techniques  are  employed,  the  derived  ratio  judgments  are 
not  necessarily  identical  to  those  obtained  from  direct  ratio  judgments.  In 
particular,  they  are  not  necessarily  integer  values  (there  is  no  way  to  divide 
100  units  to  obtain  a  5:1  ratio),  and  they  are  likely  to  yield  extreme  ratios 
near  the  end  points  (e.g.,  100  units  may  yield  ratios  as  high  as  99  and  49) 
and  more  densely  clustered  ratios  elsewhere.  Because  the  scale  values  and. 
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consequently,  the  two  inconsistency  measures  are  sensitive  to  the  response  scale 
(see  Saaty,  1980),  we  plan  to  compare  the  two  scaling  procedures  for  the  "graphi¬ 
cal  constant  sum"  method  as  well. 

Method 

A  Monte  Carlo  study  was  conducted  to  compare  the  AHP  and  GM  procedures  for 
scaling  ratio  judgments.  Thirty  different  conditions  were  generated  by  factori- 
ally  combining  three  independent  variables: 

1.  The  number  of  stimuli  to  be  scaled:  n  =  4,6,8,10,12. 

2.  The  experimental  method  for  eliciting  ratio  judgments:  direct  estimation 
and  constant  sum. 

3.  The  number  of  different  responses  allowed:  k  =  17,25,99. 

The  third  factor  of  the  present  design  reflects  an  assumption  regarding  the 
level  of  differentiation  and  precision  that  the  judge  may  achieve  when  selecting 
a  particular  response.  For  the  direct  estimation  method  the  three  values  of  k 
imply  that  the  integers  1-9  (as  suggested  by  Saaty),  1-13,  1-50  and  their  reci¬ 
procals  are  used,  respectively.  For  example,  if  k  =  17  and  the  (i,j)  cell  of  R 
is  considered,  an  integer  x  is  chosen  randomly  from  the  integers  1,...,9  such 

that  r.  =  x  and  r . .  =  1/x. 

1 » ^  ij 

When  the  constant  sum  method  is  employed,  it  is  assumed  that  the  judge  can 
divide  the  total  number  of  units  specified  by  the  experimenter  (or  the  total 
length  of  a  line)  in  only  a  finte  number  (k)  of  equally  spaced  categories.  Each 
response  is  assumed  to  be  the  mid-point  of  an  interval  of  unit  length.  The  func¬ 
tion  relating  the  numerically  calculated  ratios  r„  to  the  nonnumerical  responses 
is  convex.  For  example,  suppose  a  line  is  divided  into  k=17  equally  spaced 
intervals.  If  the  judge  places  the  marker  on  the  9th  interval,  r  =  8. 5/8. 5  =  1. 
A  marker  placed  on  the  11th  interval  yields  a  ratio  of  10.5/6.5  =  1.615,  and  one 
placed  on  the  right-hand  (17th)  interval  yields  the  highest  possible  ratio  of 
16.5/0.5  =  33.  The  first  two  levels  of  the  third  factor  (k=17,  25)  are  represen¬ 
tative  of  the  constrained  response  scales  used  in  the  psychological  literature. 

The  third  level  (k=99)  is  included  as  a  reasonable  approximation  to  the  uncon¬ 
strained  situation  in  which  the  judge  may  select  any  real  number  in  making  his 
or  her  judgment. 

For  each  of  the  30  conditions  in  the  3-way  factorial  design  described  above, 
1000  matrices  were  generated  by  independently  choosing  n(n-l)/2  uniformly  distri¬ 
buted  integers  within  the  range  of  values  dictated  by  k  and  the  method  for 
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eliciting  ratio  judgments.  The  numbers  were  randomly  placed  in  the  cells  of 
an  n  x  n  matrix  (but  excluding  the  diagonal  entries,  which  are  all  l's),  and 
their  reciprocal  values  were  assigned  to  the  corresponding  transposed  positions. 
For  each  matrix  (q=l, . . . , 1000)  in  each  of  the  30  conditions  four  solutions 
were  obtained: 

1.  The  geometric  means  of  R  's  rows:  s  , 

q  -gq 

2.  The  right  eigenvector  of  R  :  s  , 

q  -rq 

3.  The  left  eigenvector  of  R  :  s.  , 

q  -£q 

4.  The  geometric  mean  of  the  right  and  left  eigenvectors  of  R  :  s 

q  -mq 

Only  the  first  and  fourth  solutions  are  invariant  under  transposition  of  R^. 

The  estimated  scale  values  under  each  model  and  the  two  measures  of  inconsis¬ 
tency  y  and  S2  were  then  computed  and  printed. 

Results 

Table  2  presents  a  summary  of  several  characteristics  of  the  sampling 
distributions  of  y  and  S2  for  the  direct  estimation  method.  For  both  statistics, 
each  row  of  Table  2  displays  two  measures  of  central  tendency  (mean  and  median), 
a  measure  of  variability  (standard  deviation),  and  a  measure  of  skewness  defined 
as  (e.g.,  Nichols  &  Gibbons,  1979): 

e  =  3  (Mean  -  Median)  /  standard  deviation, 
e  takes  on  values  between  -3  and  3;  it  is  centered  around  0  when  the  distribu¬ 
tion  is  symmetric. 

Table  3  shows  a  similar  summary  for  the  constant  sum  method  of  eliciting 
ratio  judgments. 

Both  Tables  2  and  3  show  that  the  two  location  parameters  for  y  increase 
in  both  k  and  n,  whereas  for  S2  they  only  increase  in  k.  For  a  given  k,  and 
regardless  of  which  method  is  used  to  elicit  responses,  y  increases  on  the 
average  as  the  number  of  stimuli  grows,  whereas  S2  remains  unchanged.  For  both 
y  and  S2  and  for  both  methods,  the  standard  deviations  decrease  in  n  and,  with 
some  exceptions  in  Table  2,  increase  in  k.  Inspection  of  both  the  variability 
and  skewness  measures  shows  that  all  60  sampling  distributions  become  more 
stable  (i.e.,  less  variable)  and  symmetric  as  the  number  of  stimuli  increases. 

A  comparison  of  Tables  2  and  3  demonstrates  that  both  effects  are  stronger  for 
the  direct  estimation  case. 
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Table  2 

Summary  Statistics  of  the  Sampling  Distributions 
of  u  and  S'  for  the  Direct  Estimation  Method 


Mean 


Median 


S.D. 


Skewness 


Mean 


S2 

Median  S.D. 


Skewness 


>4 


Table  3 


Summary  Statistics  of  the  Sampling  Distributions 
of  y  and  S2  for  the  Constant  Sum  Method 


$ 
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To  Illustrate  the  increasing  stability  of  the  sampling  distributions  as 
the  number  of  stimuli  increases.  Figure  1  portrays  10  sampling  distributions 
under  the  direct  estimation  for  k*17  (the  response  scale  advocated  by  Saaty) 
and  n  ■  4,  6,  8,  10,  and  12.  The  right  panel  of  the  figure  displays  the  sam¬ 
pling  distributions  for  the  AHP  and  the  left  panel  for  the  GM  procedure.  The 
reduction  in  variability  and  skewness  is  evident  in  the  figure.  Tables  4  and 
5  present  the  critical  values  of  the  null  sampling  distributions  (at  a  =  0.10, 
0.05,  and  0.01)  for  testing  the  null  hypothesis  of  randomly  generated  responses 
Because  both  y  and  S  vanish  when  the  ratio  judgments  are  consistent,  the  null 
hypothesis  is  rejected  (and  the  derived  scale  values  may  be  safely  accepted 
and  properly  interpreted)  at  a  given  level  whenever  a  value  smaller  than  that 
listed  in  the  appropriate  table  is  obtained. 

Because  Tables  4  and  5  are  most  important  for  practical  purposes,  it  is 
desirable  to  generalize  their  results  to  values  of  n  and  k  other  than  those 
examined  in  the  present  study.  We  have  achieved  this  purpose  by  fitting  rela¬ 
tively  simple  multiple  regression  equations  based  on  n,  k,  or  some  monotonic 
transformations  of  these  two  parameters.  For  the  constant  sum  method,  the 
critical  values  of  the  sampling  distribution  of  U  are  approximated  by 
F  1Q  =  -1.2043  +  0.0979n  +  0.6719  log(n)  +  0.0008k, 

F  Q5  =  -0.9728  +  0.1150n  +  0.4243  log(n)  +  0.0003k, 

F  Q1  =  -0.4143  +  0 . 1 715n  =  0.1815  log(n), 

whereas  the  critical  values  for  the  sampling  distribution  of  S2  are  given  by: 

F  1Q  =  -2.7024  -  0 . 192 3n  +  2.9253  log(n)  +  0.0006k, 

F  Q5  =  -2.6871  -  0 . 1 351n  +  2.5866  log(n)  +  0.0003k, 

F  Q1  =  -1.8826  +  0.0411n  +  1.3067  log(n)  +  0.0006k. 

The  six  equations  above  fit  the  Monte  Carlo  results  extremely  well;  they  all 
have  adjusted  squared  multiple  correlations  of  at  least  0.99,  and  all  residuals 
are  less  than  0.085. 

For  the  direct  estimation  procedures,  the  critical  values  of  y  are  approxi 
mated  by: 

F  1Q  =  -3.1870  +  0 . 9711n  -  0.0325k  -  0.0605n2  +  0.0109nk 

F  Q5  =  -2.1582  +  0 . 62 70n  -  0.0355k  -  0.0429nz  +  0.0144nk, 

F  Q1  -  -0.4382  +  0.1744n  -  0.0511k  -  0.0155n2  +  0.0116nk. 

And  the  critical  values  for  S2  are  approximated  by 

F  1Q  -  -4.5034  +  1.0793n  +  0.0688k  -  0.0620n2  +  0.0062nk  -  0.0005k2, 

F  05  -  -4.2697  +  1.0044n  +  0.0544k  -  0.0567n2  +  0.0065nk  -  0.0005k2, 

F  Q1  =  -3.3187  +  0 . 7788n  +  0.0190k  -  0.0434n2  +  0.0077nk  -  0.0003k2. 
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The  fit  of  these  six  equations  is  not  as  impressive  as  the  fit  reported 
above  for  the  constant  sum  method.  All  the  adjusted  squared  multiple  corre¬ 
lations  exceed  0.97,  but  some  of  the  residuals  for  the  case  k=99  seem  larger 
than  desired. 

The  agreement  between  y  and  S2  is  summarized  by  several  measures  in 
Table  6.  The  two  columns  labelled  T,  one  for  each  method,  report  the  values 
of  Kendall's  rank  correlation  between  the  1000  pairs  of  y  and  S2  calculated 
separately  for  each  of  the  30  conditions.  It  is  well  known  (Marascuillu  & 
McSweeny,  1977)  that  T  can  be  provided  a  probabilistic  interpretation:  it  is 
the  difference  between  the  probabilities  of  finding  a  concordant  and  a  dis¬ 
cordant  pairs  of  values  under  random  sampling.  The  other  six  columns  focus 
on  the  regions  of  rejection  defined  in  Tables  4  and  5.  Specifically,  they 
present  the  percentage  of  cases  for  which  the  null  hypothesis  of  randomly 
generated  judgments  is  rejected  by  S2  and  y  for  a  =  0.10,  0.05,  and  0.01. 

It  can  safely  be  concluded  from  Table  6  that  the  agreement  between  y 
and  S2  is  quite  high;  the  mean  rank  order  correlation  between  y  and  S2  is 
0.813  for  the  direct  estimation  and  0.832  for  the  constant  sum  method,  and 
the  corresponding  mean  percentage  of  agreement  at  a  =  0.05  is  81  and  86, 
respectively.  Table  6  shows  that  the  two  measures  of  inconsistency  are  in 
closer  agreement  in  the  constant  sum  method.  This  si ight  advantage  of  the 
constant  sum  over  the  direct  estimation  method  is  attributed  to  the  results 


obtained  for  k=99.  Table  6  further  shows  a  strong  effect  of  the  number  of 

stimuli — as  n  increases  the  agreement  between  y  and  S2  declines. 

Following  our  examination  of  the  agreement  between  y  and  S2,  we  turn 

next  to  investigate  the  relations  between  the  four  solutions  to  the  scaling 

problem:  S  ,  S„,  S  ,  and  S  . 

-g’  -i  -r  -m 

Tables  7  and  8  present  the  median  Pearson  product  moment  correlations 


between  any  two  solutions  across  the  1000  replications.  Table  7  shows  the 
median  correlations  for  the  direct  estimation  method,  whereas  the  corresponding 
results  for  the  constant  sum  method  are  presented  in  Table  8.  All  the  corre¬ 
lations  are  high,  indicating  that  the  four  solutions  yield  similar  scale 
values  in  the  null  case  as  well  as  in  the  unidimensional  perturbed  case 
(Williams  &  Crawford,  1980).  An  examination  of  the  two  tables  reveals  several 
effects.  Here  again,  as  in  Table  6,  there  is  a  strong  and  consistent  inverse 
relationship  between  the  number  of  stimuli  (n)  and  the  degree  of  association 
between  the  various  pairs  of  solutions.  Also  similar  to  Table  6,  there  are 


Table  7 


Median  Correlat ions*  Between  the  Four 
Solutions  for  the  Direct  Estimation  Method 


S  xS 

m  r 


Decimal  points  are  omitted. 

no  systematic  differences  among  the  correlations  due  to  the  response  scale  (k). 

A  comparison  of  Tables  7  and  8  shows  that  under  the  direct  estimation  method 
the  correlations  between  the  solutions  are  slightly  higher  than  under  the  con¬ 
stant  sum  method. 

When  the  six  correlations  within  a  condition  (row)  are  compared  to  one 
another  in  either  of  the  two  tables,  an  interesting  pattern  emerges.  In  all  the 
30  cases  examined  the  lowest  correlation  is  between  the  left  and  right  eigenvec¬ 
tors  (S^*S  ),  and,  with  the  exception  of  k*=99  in  Table  7,  the  highest  correlation 
is  between  the  geometric  mean  solution  and  the  geometric  mean  of  the  two  eigen¬ 
vector  solutions  (S  *S  ).  It  mav  be  recalled  that  of  the  four  solutions,  only 
-g  -m 

S  and  S  are  invariant  under  transposition  of  the  judgment  matrix  R. 
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Table  8 


Median  Correlations*  Between  the  Four 


Solutions  for  the  Constant  Sum  Method 

k 

n 

Vs  i 

S  xS 
g  m 

S  xS 
g  r 

s0xs 

Jc  m 

S£xSr 

S  xS 
m  r 

4 

987 

996 

988 

993 

977 

996 

6 

917 

970 

913 

945 

789 

958 

17 

8 

852 

940 

834 

896 

627 

922 

10 

814 

922 

791 

881 

538 

892 

12 

765 

895 

754 

864 

479 

867 

4 

986 

996 

986 

994 

975 

995 

6 

91  4 

9  72 

915 

943 

789 

962 

25 

8 

855 
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844 

605 

10 

808 

909 
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869 

535 

899 

12 

768 
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735 
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431 

868 

4 

926 

976 

934 

945 

845 
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6 

918 
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941 

788 
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99 

8 

834 

931 

822 

876 

586 

927 

10 

791 

896 

745 

824 

479 

903 

12 

751 

852 

663 

791 

869 

876 

* 

Decimal  points  are  omitted. 


Discussion 

The  results  of  the  present  study  must  be  discussed  from  two  perspectives, 
the  practical  and  the  methodological.  At  the  practical  level,  the  results 
provide  a  much  needed  service  to  those  who  employ  ratio  scaling  procedures 
routinely  in  their  work  and  who  are  often  concerned  with  the  reliability  and 
consistency  of  their  data.  The  results  presented  in  Tables  2  through  5  provide 
these  researchers  a  sound  basis  for  detecting  inconsistent  judgments  and 
subjects.  They  supply  decision  rules  for  accepting  or  rejecting  judgment 
matrices  for  both  the  AHP  and  GM  procedures.  In  the  former  case,  we  contend 
that  use  of  the  traditional  approach  to  hypothesis  testing,  in  which  the  null 
hypothesis  is  rejected  with  a  predetermined  probability  of  type  I  error,  is 
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superior  to  any  of  the  rules  of  thumb  advocated  by  Saaty.  In  the  latter  case, 
our  results  fill  a  gap  in  the  long  line  of  psychometric  studies  on  the  proper¬ 
ties  of  the  geometric  means. 

Like  any  other  simulation,  our  results  are  limited  to  those  combinations 
of  parameters  tested.  As  such,  they  lack  the  generality  achievable  by  a  theore¬ 
tical  development.  However,  the  values  of  n  and  k  employed  in  the  present  study 
are  representative  of  the  choices  usually  encountered  in  applications,  so  that 
many  users  should  find  our  tables  useful.  Those  using  different  response  scales 
and  a  larger  number  of  stimuli  will  benefit  from  the  approximation  formulas 
developed  for  the  critical  points  of  the  sampling  distributions.  Undoubtedly, 
these  formulas  may  be  further  refined  in  similar  studies  coveming  a  larger 
range  of  parameter  values. 

We  point  out  to  the  more  conservative  user  that  the  results  in  Tables  2 
and  3  can  be  used  to  approximate  the  mean  and  variance  of  the  sampling  distribu¬ 
tions  which,  in  conjunction  with  Tchebychef f ' s  inequality,  could  be  used  to 
determine  more  conservative  decision  rules.  For  example,  in  the  direct  estima- 
tion  method  the  mean  and  standard  deviation  of  S  can  be  approximated  by 

S2  =  1.2259  +  0 . 0129n  +  0.0825k, 

o(S2)  =  [2.440  -  0 . 335n  +  0.006k  +  0.016n  -  O.OOlnk]  . 

Both  equations  fit  the  data  well  (R^  >  0.975).  Similar  approximations  can 

be  generated  for  other  cases. 

At  a  more  general  methodological  level,  we  conclude  that  there  is  a  very 
high  level  of  agreement  between  the  AHP  and  GM  procedures  in  the  null  case.  We 
base  this  conclusion  on  high  correlations  between  the  estimated  scale  values 
(Tables  7  and  8),  the  large  concordance  between  the  rank  orderings  of  the 
various  matrices  by  means  of  y  and  S2,  and  the  high  agreement  between  the  two 
rejection  rules  for  various  levels  of  a  (Table  6).  It  was  pointed  out  in  the 
introduction  that  one  interpretat ional  problem  of  the  AHP  stems  from  its  lack 
of  invariance  under  transposition  of  R.  The  magnitude  of  this  problem  is  well 
documented  in  Tables  7  and  8,  where  the  right  and  left  eigenvectors  are  shown 
to  intercorrelate  lower  than  any  other  pair  of  solutions.  This  finding  rein¬ 
forces  the  warning  of  Johnson,  Beine,  and  Wang  (1979)  regarding  the  use  of  the 
right  eigenvector.  It  is  reassuring  to  note  that  in  the  same  two  tables  the 
highest  correlations  are  typically  those  between  the  two  invariant  solutions. 
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When  our  results  are  considered  in  conjunction  with  the  findings  of 
Williams  and  Crawford  (1980)  and  the  objection  raised  by  Johnson,  Beine,  and 
Wang  (1979),  a  strong  case  can  be  made  in  favor  of  the  GM  as  a  better  and 
more  convenient  procedure  than  the  AHP  for  scaling  pairwise  ratio  judgments. 
In  the  null  case  as  well  as  in  the  consistent  case,  the  two  procedures  are 
practically  indistinguishable.  But  the  geometric  means  are  easier  to  calcu¬ 
late  and  interpret,  and  they  recover  the  "true"  scale  values  with  higher 
accuracy  in  the  presence  of  measurement  error. 
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